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Preface  &  Acknowledgements 


Welcome  to  our  Tenth  Annual  Acquisition  Research  Symposium!  We  regret  that  this 
year  it  will  be  a  “paper  only”  event.  The  double  whammy  of  sequestration  and  a  continuing 
resolution,  with  the  attendant  restrictions  on  travel  and  conferences,  created  too  much 
uncertainty  to  properly  stage  the  event.  We  will  miss  the  dialogue  with  our  acquisition 
colleagues  and  the  opportunity  for  all  our  researchers  to  present  their  work.  However,  we 
intend  to  simulate  the  symposium  as  best  we  can,  and  these  Proceedings  present  an 
opportunity  for  the  papers  to  be  published  just  as  if  they  had  been  delivered.  In  any  case,  we 
will  have  a  rich  store  of  papers  to  draw  from  for  next  year’s  event  scheduled  for  May  14-15, 
2014! 


Despite  these  temporary  setbacks,  our  Acquisition  Research  Program  (ARP)  here  at 
the  Naval  Postgraduate  School  (NPS)  continues  at  a  normal  pace.  Since  the  ARP’s 
founding  in  2003,  over  1 ,200  original  research  reports  have  been  added  to  the  acquisition 
body  of  knowledge.  We  continue  to  add  to  that  library,  located  online  at 
www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  70  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  encourage  your  future  participation. 

Unfortunately,  what  will  be  missing  this  year  is  the  active  participation  and 
networking  that  has  been  the  hallmark  of  previous  symposia.  By  purposely  limiting 
attendance  to  350  people,  we  encourage  just  that.  This  forum  remains  unique  in  its  effort  to 
bring  scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  It  provides  the  opportunity  to  interact  with  many  top  DoD 
acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both  in  the  formal 
panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals,  breaks,  and  the 
day-ending  socials.  Many  of  our  researchers  use  these  occasions  to  establish  new  teaming 
arrangements  for  future  research  work.  Despite  the  fact  that  we  will  not  be  gathered 
together  to  reap  the  above-listed  benefits,  the  ARP  will  endeavor  to  stimulate  this  dialogue 
through  various  means  throughout  the  year  as  we  interact  with  our  researchers  and  DoD 
officials. 

Affordability  remains  a  major  focus  in  the  DoD  acquisition  world  and  will  no  doubt  get 
even  more  attention  as  the  sequestration  outcomes  unfold.  It  is  a  central  tenet  of  the  DoD’s 
Better  Buying  Power  initiatives,  which  continue  to  evolve  as  the  DoD  finds  which  of  them 
work  and  which  do  not.  This  suggests  that  research  with  a  focus  on  affordability  will  be  of 
great  interest  to  the  DoD  leadership  in  the  year  to  come.  Whether  you’re  a  practitioner  or 
scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 
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•  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  & 
Logistics) 

•  Director,  Acquisition  Career  Management,  ASN  (RD&A) 

•  Program  Executive  Officer,  SHIPS 

•  Commander,  Naval  Sea  Systems  Command 

•  Program  Executive  Officer,  Integrated  Warfare  Systems 

•  Army  Contracting  Command,  U.S.  Army  Materiel  Command 

•  Office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 

•  Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics,  & 
Technology) 

•  Deputy  Director,  Acquisition  Career  Management,  U.S.  Army 

•  Office  of  Procurement  and  Assistance  Management  Headquarters, 
Department  of  Energy 

•  Director,  Defense  Security  Cooperation  Agency 

•  Deputy  Assistant  Secretary  of  the  Navy,  Research,  Development,  Test,  & 
Evaluation 
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•  Director,  Office  of  Acquisition  Resources  and  Analysis  (ARA) 

•  Deputy  Assistant  Secretary  of  the  Navy,  Acquisition  &  Procurement 

•  Director  of  Open  Architecture,  DASN  (RDT&E) 

•  Program  Executive  Officer,  Littoral  Combat  Ships 
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Abstract 

As  directed  by  the  National  Defense  Authorization  Act  (NDAA)  of  2010,  Public  Law  1 11-84, 
the  defense  acquisition  community  is  transitioning  in  an  effort  to  adopt  software  best 
practices  for  delivering  information  technology  in  an  incremental  and  iterative  model.  The 
Deputy  Secretary  of  Defense  provided  a  report  to  Congress  titled  A  New  Approach  for 
Delivering  Information  Technology  Capabilities  in  the  DoD,  delineating  the  overarching 
framework  to  reform  the  acquisition  of  information  technology  to  better  address  and  fulfill 
warfighter  requirements.  Many  governmental  agencies,  anticipating  future  directives,  are 
implementing  Agile  software  development  methodologies  and  demonstrating  success  using 
these  methodologies  on  DoD-sponsored  programs.  As  an  example  of  this,  the  Rapid 
Integration  and  Test  Environment  (RITE)  established  by  SSC  Pacific  in  2008  provides  a 
standardized  Agile  development  environment  for  its  C2  programs.  Much  of  the  work  to  date 
has  addressed  program  items  controlled  at  lower  command  levels  while  awaiting 
restructuring  of  the  acquisition  milestone  and  review  requirements  specified  in  DoDI  5000.02. 
This  report  presents  the  research  completed  in  analyzing  traditional  acquisition  program 
milestone  reviews  and  documentation  requirements  and  identifies  streamlining  opportunities 
that  support  Agile  development.  The  report  also  validates  the  RITE  initiative  in  providing  the 
structured  engineering  approach  that  makes  Agile  development  viable  in  a  DoD  acquisition 
environment. 

Introduction 

The  National  Defense  Authorization  Act  (NDAA)  for  Fiscal  Year  2010,  Public  Law 
1 11-84,  Section  804 — hereafter  referred  to  as  Sec.  804,  2010  NDAA — established  the 
requirement  for  the  Department  of  Defense  (DoD)  to  streamline  the  acquisition  of 
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information  technology.  In  response  to  that  request,  the  Office  of  the  Deputy  Secretary  of 
Defense  (2010)  provided  a  report  titled  A  New  Approach  for  Delivering  Information 
Technology  Capabilities  in  the  DoD.  This  report  created  the  overarching  framework  to 
reform  the  acquisition  of  information  technology  to  better  address  and  fulfill  warfighter 
requirements.  While  this  new  requirement  established  the  basics  for  streamlining  information 
technology  acquisition,  it  did  little  to  provide  meaningful,  actionable  practices  that  an 
acquisition  program  can  execute.  The  goal  of  this  research  was  to  identify  opportunities  to 
create  actionable  Agile  processes  that  information  technology  programs  can  use  to  execute 
streamlined  programs. 

Background 

The  Sec.  804,  2010  NDAA  requirement  established  the  parameters  for  the  new 
acquisition  process  based  on  the  March  2009  report  of  the  Defense  Science  Board  (DSB) 

T ask  Force  titled  Department  of  Defense  Policies  and  Procedures  for  the  Acquisition  of 
Information  Technology.  The  report  was  required  to  include  several  characteristics  that 
Congress  determined  necessary  for  successful  implementation: 

1 .  early  and  continual  involvement  of  the  user; 

2.  multiple,  rapidly  executed  increments  or  releases  of  capability; 

3.  early,  successive  prototyping  to  support  an  evolutionary  approach;  and 

4.  a  modular,  open-systems  approach.  (NDAA  for  Fiscal  Year  2010,  2009) 

These  characteristics  are  significant  in  that  they  also  describe  the  elements 
indicative  of  an  Agile  development  methodology. 

In  response  to  Sec.  804,  2010  NDAA,  the  DoD  provided  a  report  to  Congress 
highlighting  its  plans  to  reinvent  the  IT  acquisition  process.  Noting  the  departure  necessary 
from  a  traditional  acquisition  process,  the  DoD  provided  the  following: 

Acquisition  activities  in  the  new  process  for  delivering  IT  capability  will  differ 
significantly  from  the  traditional  weapon  system  development  acquisition  process  and  will  be 
separately  defined  in  DoD  IT  acquisition  policy  issuances.  The  IT  acquisition  process  will  be 
agile  to  respond  to  a  dynamic  technology  environment  and  to  address  unique  challenges, 
such  as  cyber  threats  (Office  of  the  Deputy  Secretary  of  Defense,  2010,  p.  9). 

As  shown  in  the  next  section,  this  approach  provides  a  flexible  structure  dedicated  to 
positive,  customer-driven  outcomes. 

Agile  Development 

Agile  development  focuses  on  close  customer  interaction  and  rapid,  iterative,  and 
incremental  development  cycles  that  produce  a  working  product.  This  approach  focuses  on 
early  feedback  and  flexibility  adapting  to  customer  needs. 

In  describing  Agile  methods,  Lapham  et  al.  (2011)  noted  that  the  concepts  and 
practices  associated  with  Agile  development  arose  out  of  the  Agile  Alliance.  In  an  effort  to 
identify  an  alternative  to  elaborate  and  time-consuming  software  development  processes, 
the  Agile  Alliance  created  a  set  of  values  that  focus  on  people,  collaboration,  and 
development  of  quality  software  products  for  their  customers  (Lapham  et  al.,  201 1 ,  p.  1 ). 

The  Agile  Alliance’s  efforts  resulted  in  the  Agile  Manifesto  for  Agile  Software 
Development: 

We  are  uncovering  better  ways  of  developing  software  by  doing  it  and  helping 
others  do  it.  Through  this  work  we  have  come  to  value: 
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Individuals  and  interactions  over  processes  and  tools 
Working  software  over  comprehensive  documentation 
Customer  collaboration  over  contract  negotiation 
Responding  to  change  over  following  a  plan 
That  is,  while  there  is  value  in  the  items  on  the  right,  we  value  the  items  on 
the  left  more.  (Lapham  et  al.,  201 1 ,  p.  1  )1 

Critics  of  Agile  development  cite  documentation  reduction  as  problematic  in 
development  efforts,  but  these  concerns  are  discounted  by  seasoned  developers.  In  Agile 
development,  the  amount  of  documentation  is  determined  by  the  software,  not  the  desire  of 
the  developer.  It  is  essential  to  understand  that  while  documentation  is  important,  it  should 
not  act  as  a  replacement  for  communication  and  collaboration.  Regarding  Agile 
development’s  approach  to  documentation,  Lapham,  Williams,  Hammons,  Burton,  and 
Schenker  (2010)  observed,  “The  Agile  community  would  argue  instead  that  documentation 
is  important,  but  no  more  documentation  should  be  created  than  is  absolutely  necessary  to 
support  the  development  itself  and  future  sustainment  activities”  (p.  4).  Documentation 
developed  using  the  Agile  methodology  can  support  the  intent  and  objectives  of  the 
documentation  requirements  of  the  DoD  acquisition  process. 

Agile  development  is  not  the  only  initiative  working  to  streamline  and  improve  the 
effectiveness  of  development  activities.  The  Space  and  Naval  Warfare  Systems  Command 
(SPA WAR)  Rapid  Integration  and  Test  Environment  (RITE)  initiative  focused  their  efforts  on 
key  areas  in  the  development  cycle  that  work  collectively  to  shorten  cycle-time  and  improve 
the  efficiency  of  the  development  effort. 

Rapid  Integration  and  Test  Environment 

In  2008,  the  Program  Executive  Office  (PEO)  Command,  Control,  Communications, 
Computers,  and  Intelligence  (C4I),  Command  and  Control  Program  Office  (PMW  150) 
began  implementation  of  the  RITE  initiative.  This  initiative  was  born  out  of  necessity  in  that 
the  existing  process  for  requirements  definition  and  management,  as  well  as  processes  for 
software  development,  did  not  consistently  deliver  high-quality  Navy  Command  and  Control 
(C2)  systems  either  on  time  or  within  budget. 

The  RITE  initiative,  as  implemented,  represents  a  new  life  cycle  model  for  Navy  C2 
software  that  meets  many  of  the  process  objectives  identified  in  Sec.  804,  2010  NDAA  and 
improves  efficiencies  in  Navy  C2  application  development.  RITE  places  increased  emphasis 
on  early  and  frequent  customer  interaction  and  software  testing,  as  well  as  necessary 
software  engineering  practices  at  the  source  code  level.  RITE  is  a  structured  approach  to 
software  development,  taking  full  advantage  of  technology  advances  and  open-source 
models  to  automate  processes  and  shorten  development  cycles — thereby  increasing  the 
maintainability  of  the  software  baselines.  The  new  automated  processes  also  allow  a 
reduction  in  low-value-added  processes  and  manually  developed  reports,  further 
streamlining  the  acquisition  cycle  and  improving  efficiencies.  The  initiative  clarifies  software 
delivery  requirements,  adds  additional  engineering  rigor  to  deliverables,  and  reduces  the 
opportunity  for  misunderstanding  between  end  users  and  developers.  Lastly,  RITE  uses  a 
centralized  information  repository  that  allows  all  stakeholders  to  communicate,  coordinate, 
and  collaborate  virtually. 


1  The  Manifesto  for  Agile  Development  was  created  during  a  meeting  of  representatives  from  across 
the  nascent  Agile  community  and  included  the  following:  Kent  Beck,  Mike  Beedle,  Arie  van 
Bennekum,  Alistair  Cockburn,  Ward  Cunningham,  Martin  Fowler,  James  Grenning,  Jim  Highsmith, 
Andrew  Hunt,  Ron  Jeffries,  Jon  Kern,  Brian  Marick,  Robert  C.  Martin,  Steve  Mellor,  Ken  Schwaber, 
Jeff  Sutherland,  and  Dave  Thomas. 
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As  RITE  has  evolved  and  process  improvements  have  been  realized,  additional  uses 
for  RITE  in  support  of  the  C2  life  cycle  have  been  identified.  This  support  includes  facilitating 
close  collaboration  with  outside  agencies  to  ensure  that  the  development  knowledge  and 
test  and  evaluation  (T&E)  results  are  shared  in  order  to  reduce  overall  project  time.  Figure  1 
shows  the  RITE  processes  as  they  align  with  all  four  phases  of  the  new  IT  acquisition  life 
cycle.  The  arrows  indicate  areas  where  RITE  (consisting  of  people,  processes,  and 
infrastructure)  directly  supports  the  acquisition  of  Navy  C2  capabilities  and  systems. 


Figure  1.  RITE  Alignment  With  2010  IT  Acquisition  Changes 


Defense  Acquisition  Management  System 

The  Defense  Acquisition  Management  System  (see  Figure  2)  is  the  management 
process  guiding  all  DoD  acquisition  programs.  The  initiating  directive,  DoD  Directive  (DoDD) 
5000.01,  provides  the  policies  and  principles  that  govern  the  defense  acquisition  system, 
and  DoD  Instruction  (DoDI)  5000.02,  Operation  of  the  Defense  Acquisition  System,  provides 
the  management  framework  that  implements  these  policies  and  principles.  “The  Defense 
Acquisition  Management  Framework  provides  an  event-based  process  where  acquisition 
programs  progress  through  a  series  of  milestones  associated  with  significant  program 
phases”  (DoD,  2012). 

The  Defense  Acquisition  Management  System  is  used  throughout  the  DoD  as  the 
single  overarching  methodology  for  acquiring  business  and  weapons  systems. 
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Figure  2.  The  Defense  Acquisition  Management  System 
Related  Research 


Defense  Science  Board  Task  Force  Report  on  Department  of  Defense  Policies  and 
Procedures  for  the  Acquisition  of  Information  Technology 

In  March  2009,  the  DSB  Task  Force  reported  on  the  evaluation  of  the  acquisition  of 
information  technology  (IT)  within  the  DoD.  This  report  identified  critical  problems  with  the 
management  of  IT  acquisitions  using  an  enterprise  approach  resulting  in  a  “profound 
operational  impact”  (DSB  Task  Force,  2009,  p.  1).  The  report  identified  problems  in 
responsiveness  and  the  ability  to  address  operational  needs.  Citing  a  2006  DSB  study  titled 
Information  Management  for  Net  Centric  Operations,  the  report  noted, 

Especially  important,  according  to  the  2006  report,  was  that  much  of  the 
military  capability  used  to  support  the  conflicts  was  paid  with  supplemental 
funding — programs  that  were  not  part  of  the  Department’s  planned  capability. 
This  circumstance  reflects  the  fact  that  the  need  for  such  programs  could  not 
be  predicted  during  previous  core  program  and  budget  planning,  and  the 
system  was  not  sufficiently  agile  to  react  once  the  need  was  apparent.  (DSB 
Task  Force,  2009,  pp.  1-2) 

The  report  goes  on  to  identify  the  evolution  of  weapons  system  software  reliance  in 
the  1970s  at  20%  to  as  much  as  80%  in  2000.  This  is  a  critical  issue  in  light  of  the  reduction 
in  U.S.  computing  graduates  and  qualified  expert  government  staff  and  increased  reliance 
on  IT  at  a  time  of  rising  vulnerabilities  and  threats  (see  Figure  3;  DSB  Task  Force,  2009,  p. 
6). 
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Figure  3.  The  Perfect  IT  Storm 

(DSB  Task  Force,  2009) 


The  DSB  Task  Force’s  findings  identified  the  need  for  a  unique  acquisition  process 
for  IT.  Commenting  on  the  failure  of  major  defense  systems,  the  task  force  also  identified 
the  need  to  shorten  the  lengthy  acquisition  process  and  to  provide  the  flexibilities  necessary 
to  support  continuous  changes  and  upgrades.  Other  critical  elements  of  change  identified 
by  the  DSB  Task  Force  include  the  need  to  align  acquisition  authorities  and  organizational 
structure  under  the  Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology, 
and  Logistics  (OUSD[AT&Lj)  to  better  manage  the  technical  aspects  of  IT  acquisitions  and 
the  need  to  consider  proven  experience  as  an  added  component  in  evaluating  the  education 
and  certification  of  members  of  the  acquisition  workforce. 

Considerations  for  Using  Agile  in  DoD  Acquisition  (Carnegie  Mellon  University, 
Software  Engineering  Institute) 

This  document  was  created  to  provide  additional  information  on  Agile  development 
as  it  relates  to  DoD  acquisitions,  references  actual  DoD  programs  that  have  benefited  from 
the  adoption  of  Agile  practices  within  their  respective  programs,  and  includes  analysis  of 
relevant  literature  regarding  Agile  development.  Lapham  et  al.  (2010)  answered  many 
questions  regarding  Agile  development,  but  they  specifically  answered  whether  Agile 
development  methods  are  able  to  produce  better  products  within  cost  and  schedule 
requirements  (yes)  and  addressed  the  barriers  which  inhibit  the  DoD’s  adoption  of  Agile 
development  methods. 

In  determining  the  barriers  to  DoD’s  Agile  development  adoption,  Lapham  et  al. 
(2010)  noted, 

The  barriers  to  adopting  Agile  in  the  DoD  appear  to  be  primarily  cultural.  That 
is  to  say  that  there  is  little  in  the  way  of  regulation  or  guidance  provided  in 
DoDI  5000.02  that  would  prevent  the  use  of  Agile.  This  instruction  does 
impose  specific  constraints  on  the  acquisition  office,  but  these  constraints 
would  be  true  of  any  development  environment,  (p.  27) 

While  not  finding  any  primary  barriers  within  the  DoDI  5000.02,  Lapham  et  al.  (2010) 
did  address  issues  with  the  Federal  Acquisition  Regulation,  citing  the  need  to  address 
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contracting  requirements  to  support  Agile  development.  These  changes  would  require  the 
accommodation  of  Agile  as  part  of  a  system’s  acquisition  strategy  at  the  beginning  of  a 
program  development  effort  (Lapham  et  al.,  2010,  p.  27).  The  authors  also  pointed  to 
significant  concerns  regarding  milestone  reviews  within  the  DoD  acquisition  system: 

A  very  specific  acquisition  issue  and  sticking  point  is  that  Agile  methodology 
does  not  accommodate  large  capstone  events  such  as  Critical  Design 
Review  (CDR),  which  is  usually  a  major,  multi-day  event  with  many  smaller 
technical  meetings  leading  up  to  it.  This  approach  requires  a  great  deal  of 
documentation  and  many  technical  reviews  by  the  contractor.  (Lapham  et  al., 
2010,  p.  13) 

In  addressing  the  primary  questions  raised  regarding  Agile  development  and  its  use 
within  the  DoD,  Lapham  et  al.  (2010)  noted  that  end-user  participation  and  culture  are 
issues  that  must  be  addressed  before  using  Agile  methods  within  a  program  (p.  44). 

Agile  Methods:  Selected  DoD  Management  and  Acquisition  Concerns  (Carnegie 
Mellon  University,  Software  Engineering  Institute) 

This  document  is  the  second  in  a  series  regarding  Agile  development  methods  and 
the  use  of  Agile  within  the  DoD.  While  focusing  on  a  better  understanding  of  Agile 
development  as  it  pertains  to  the  DoD  acquisition  system,  Lapham  et  al.  (201 1)  targeted  this 
report  to  address  Agile  development  implementation  approaches  for  acquisition  and 
development  personnel  (p.  2). 

Lapham  et  al.  (201 1 )  provided  thorough  discussions  of  Agile  development,  why  Agile 
methods  are  increasing  within  the  DoD,  contracting  requirements  for  implementation  within 
Agile  programs,  and  the  use  of  change  management  within  an  organization,  specifically 
applicable  to  a  program  management  office  (PMO),  to  implement  Agile  methods.  Most 
applicable  to  the  analysis  within  this  paper  is  the  discussion  of  milestone  reviews  within 
systems  development  and  its  effect  on  Agile  development.  (Lapham  et  al.,  201 1 ,  pp.  1 0-1 1 ). 
The  authors  provided  a  thorough  evaluation  of  milestone  reviews,  including  the  effort 
required  to  produce  the  supporting  documentation  and  not  the  challenges  associated  with 
adapting  a  program’s  milestone  reviews  to  an  Agile  methodology: 

The  intent  of  any  technical  milestone  review  is  for  evaluation  of  progress 
and/or  technical  solution.  For  PMOs  trained  and  experienced  in  the  traditional 
acquisition  methods,  evaluating  program  progress  and  technical  solutions 
follows  well  established  guidelines  and  regulations.  Very  specific 
documentation  is  produced  to  provide  the  data  required  to  meet  the  intent  of 
the  technical  review  as  called  out  in  the  program  specific  Contract  Data 
Requirements  List  (CDRL).  The  content  of  these  documents  and  the  entry 
and  exit  criteria  for  each  review  is  well  documented.  However,  even  in 
traditional  acquisitions  (using  traditional  methods),  these  documents,  exit  and 
entry  criteria  can  be  and  usually  are  tailored  for  the  specific  program.  Since 
the  documentation  output  from  Agile  methods  appears  to  be  “light”  in 
comparison  to  traditional  programs,  the  tailoring  aspects  take  on  additional 
aspects.  Some  of  the  specific  challenges  for  Agile  adoption  that  we  observed 
during  our  interviews  that  must  be  addressed  are  as  follows: 

•  incentives  to  collaborate, 

•  shared  understanding  of  definitions/key  concepts, 

•  document  content — the  look  and  feel  may  be  different  but  the  intent  is 
the  same — and 
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•  regulatory  language.  (Lapham  et  al.,  201 1 ,  pp.  38-39) 

Analytical  Approach 

The  analytical  approach  involved  exhaustive  analysis  of  technical  reviews  and 
documentation  to  identify  possible  areas  in  which  duplication  or  overlap  currently  exists 
within  the  review  structure  or  the  documentation  set  required  when  developing  a  product. 

The  review  included  a  thorough  analysis  of  all  milestone  reviews  and  documentation 
associated  with  a  typical  development  effort.  The  analysis  examined  the  technical  definition 
of  each  review,  the  statutory  or  regulatory  requirement  upon  which  it  is  based,  the  program 
participant/organization  responsible  for  execution  of  the  review,  the  program 
participant/organization  responsible  for  conducting  the  review/completing  the  document 
(subordinate  organization — typically  Software  Support  Activity  [SSA],  In-service  Engineering 
Agent  [ISEA],  etc.),  key  team  members  involved,  entrance  and  exit  criteria  for  the  review, 
recipient  of  the  completed  review  results  (PEO,  Milestone  Decision  Authority  [MDA],  etc.), 
any  other  stakeholders,  and  previous  and  next  process  flow  steps.  The  review  process  was 
refined  to  focus  on  the  following  milestone  reviews:  Preliminary  Design  Review  (PDR), 
Critical  Design  Review  (CDR),  Test  Readiness  Review  (TRR),  System  Verification  Review 
(SVR),  and  Production  Readiness  Review  (PRR),  which  were  evaluated  against  Agile 
development  requirements.  Further  analysis  was  conducted  against  the  DoD  and  SPAWAR 
Systems  Command  (SPAWARSYSCOM)  System  Engineering  Technical  Review  (SETR) 
PDR  and  CDR  Risk  Assessment  Checklists  to  provide  a  cross-referenced  analysis  against 
PDR  and  CDR  requirements.  These  checklists  were  targeted  due  to  their  complexity  (The 
DoD  PDR  checklist  is  860  line  items,  and  the  DoD  CDR  checklist  is  929  line  items)  and  their 
applicability  within  development  timelines  associated  with  Agile  development.  Although 
SPAWARSYSCOM  SETR  checklists  for  PDR/CDR  closely  follow  the  DoD  checklists  (with 
871  and  906  line  items,  respectively),  the  difference  in  line  items  represents  tailoring  to 
address  Navy  specific  requirements. 

The  documentation  analysis  included  an  evaluation  of  which  milestones  within  the 
defense  acquisition  system  required  completion  or  updating  of  each  specific  document. 
Additionally,  the  evaluation  included  the  review  of  the  documentation  set  required  by  the 
SPAWARSYSCOM  SETR  Risk  Assessment  Checklists. 

Results 

This  section  highlights  the  pertinent  analysis  of  the  reviews  and  documentation 
information  collected  during  the  preliminary  part  of  this  effort.  Discussions  with  experienced 
program  professionals  and  other  acquisition  workforce  personnel  also  occurred  during  the 
data  collection  and  analysis  phases  to  better  inform  the  group’s  decision-making  process. 

Of  note,  during  the  analytical  phase  of  this  effort,  discussions  regarding  the  role  of 
the  cognizant  technical  authority  (TA)  and  their  impact  (positively  or  negatively)  on  the 
viability  of  the  development  effort.  According  to  the  Naval  Warfare  Systems  Certification 
Policy,  a  TA’s  role  within  an  organization  is  as  follows: 

The  entity  with  the  authority,  responsibility,  accountability,  and  technical 
integrity  to  establish,  monitor,  and  approve  technical  standards,  tools,  and 
processes  in  compliance  with  applicable  DoD  and  DoN  policy,  requirements, 
architectures,  and  standards.  (DoN,  2012,  pp.  B-6) 

While  the  TA’s  role  is  focused  on  institutional  level  technical  compliance,  the  TA’s 
role  remains  secondary  to  the  program  manager’s  (PM’s)  and  MDA’s  role  in  validating  and 
approving  the  planned  milestone  review  and  programmatic  documentation  streamlining 
efforts.  Even  so,  the  TA’s  role  as  the  technical  advocate  in  support  of  development  methods 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


- 194- 


such  as  Agile  cannot  be  overstated.  A  TA’s  commitment  (and  through  extension,  a 
command’s  commitment)  to  Agile  development  can  be  helpful  in  supporting  the  MDA’s 
decision  to  approve  a  PM’s  request  to  eliminate  or  otherwise  minimize  documentation 
requirements. 


Primary  Review  Analysis 

The  initial  analysis  of  technical  reviews  included  the  following:  Initial  Technical 
Review  (ITR),  Alternative  System  Review  (ASR),  Integrated  Baseline  Review  (IBR),  System 
Requirements  Review  (SRR)  .Technology  Readiness  Assessment  (TRA),  System 
Functional  Review  (SFR),  PDR,  CDR,  TRR,  SVR,  Functional  Configuration  Audit  (FCA), 
PRR,  Operational  Test  Readiness  Review  (OTRR),  Physical  Configuration  Audit  (PCA), 
Integration  Readiness  Review  (IRR),  In  Service  Review  (ISR),  Development  Test  Readiness 
Review  (DTRR),  and  Operational  Test  Readiness  Review  (OTRR).  Although  this  analysis 
was  an  essential  first  step  and  helped  to  visualize  individual  reviews  within  the  context  of  the 
DoD  Acquisition  Management  System  (see  Figure  4  ),  no  major  streamlining  opportunities 
were  identified  in  the  analysis. 
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Figure  4.  System  Engineering  Technical  Reviews  According  to  the  DoD  Acquisition 

Management  System 


In  evaluating  the  reviews  against  Agile  development  principles,  it  was  evident  that  to 
achieve  any  streamlining  within  the  review  process,  the  numerous  review  requirements 
would  need  to  be  downsized  and  re-envisioned  to  address  the  primary  elements  of  the 
existing  reviews.  This  was  preliminarily  documented  in  the  DSB  Task  Force’s  (2009)  report 
Department  of  Defense  Policies  and  Procedures  for  the  Acquisition  of  Information 
Technology  (see  Figure  5).  The  DSB  Task  Force’s  (2009)  recommendation  streamlined  the 
milestone  review  process  to  eliminate  the  complex,  all-encompassing  milestone  reviews  in 
favor  of  more  frequent,  tailored  decision  points  that  enable  a  program  to  identify  problems 
earlier,  which  results  in  more  “robust  and  maintainable  designs”  (pp.  52-53). 
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Figure  5.  New  Acquisition  Process  for  Information  Technology 

(DSB  Task  Force,  2009,  p.  48) 


In  the  context  of  the  primary  milestone  reviews  (PDR,  CDR,  and  SVR/PRR),  a 
nominal  Agile  development  structure  was  created  (see  Figure  6),  providing  increment 
releases  (two-year  cycles)  that  include  service  packs  (six-month  cycles  of  completed 
development  efforts  that  have  the  potential  to  be  forwarded  as  release  candidates).  Within 
each  service  pack  is  a  series  of  sprints,  which  represent  a  standard  form  of  Agile 
development.  This  construct  allows  the  identification  of  a  Build  Review  (BR;  reviews  are 
shown  in  red  in  Figure  6)  at  the  beginning  of  each  service  pack,  which  addresses  elements 
of  the  increment  level  PDR  and  subsequent  CDR;  an  Interim  Progress  Review  (IPR)  at 
Sprint  3  or  4  to  assess  progress  regarding  cost,  schedule,  and  performance  and  evaluate 
the  service  pack  functional  backlog  compared  to  the  current  backlog,  validating  the  detailed 
design  of  the  remaining  sprints;  and  a  Fielding  Review  (FR)  at  the  end  of  the  sprint  cycle. 
These  reviews  throughout  the  sprint/service  pack  cycles  supplant  the  traditional 
PDR/CDR/SVR/PRR  reviews  and  relate  directly  to  the  decision  points  described  in  the  DSB 
Task  Force’s  (2009)  report  to  Congress,  as  shown  in  Figure  5. 2 


2  Service  pack  functional  backlog,  from  an  Agile  development  perspective,  is  a  prioritized  listing  of 
allocated  requirements  (in  Agile  terms,  stories)  determined  at  the  beginning  of  the  sprint  to  be 
sufficient  tasking  to  complete  within  the  sprint  cycle.  The  current  backlog  is  the  amount  of  the  service 
pack  functional  backlog  remaining  within  the  sprint  and  is  used  to  determine  the  progress  against  the 
planned  effort. 
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Figure  6.  Linkage  Between  DoD  Acquisition  Mangaement  System  Reviews  and  Agile 

Development  Reviews 

Given  the  potential  differences  in  the  wide  variety  of  program  development  efforts, 
tailoring  of  the  reviews  to  best  support  the  specific  aspects  of  a  program  is  necessary.  This 
customization  can,  as  indicated  previously,  be  structured  such  that  the  sum  of  the  review 
content  is  equal  to  the  sum  of  the  replaced  reviews. 

Just  as  the  reviews  themselves  are  being  streamlined,  the  supporting  documentation 
should  be  streamlined  to  eliminate  unnecessary  effort. 

Documentation  Analysis 

The  documentation  review  resulted  in  a  comprehensive  analysis  that  provides  a 
high-level  overview  of  acquisition  documentation.  Although  it  was  expected,  the  review 
verified  that  because  a  program  is  required  to  increase  reporting  responsibilities  to  address 
statutory  and  regulatory  requirements,  opportunities  for  significant  streamlining  are  greatly 
reduced.  This  is  particularly  true  for  Major  Defense  Acquisition  Programs  (MDAPs)  and 
Major  Automated  Information  Systems  (MAISs).  It  is  the  remaining  programs  that  can 
benefit  from  a  reduction  in  documentation  associated  with  regulatory  requirements; 
specifically,  small  software  intensive  development  efforts.  This  does  not  preclude  the  use  of 
Agile  development  as  a  component  of  larger  projects  (such  as  for  a  software  development 
effort  ancillary  to  a  major  hardware  development  effort),  but  it  will  require  a  significant 
amount  of  negotiation  with  the  MDA. 

In  analyzing  individual  document  requirements,  it  was  apparent  that  aggregate 
generalizations  regarding  documentation  do  little  to  support  the  tailoring  of  a  program  to 
streamline  reporting  requirements  other  than  to  say  that  it  is  possible.  As  Lapham  et  al. 
(2010)  reported, 

Those  programs  that  have  used  Agile  in  software  development  have  found 
that  the  DoD  5000  series  has  great  flexibility  and  does  not  in  fact  preclude  the 
use  of  Agile.  It  appears  that  with  careful  review  and  some  tailoring  an 
alternate  interpretation  can  be  created  so  that  Agile  can  be  used  on  DoD 
programs,  (p.  13) 

This  analysis,  while  correct  in  identifying  the  DoD  5000  series  as  the  prime  set  of 
regulatory  hurdles  with  which  to  contend,  shows  that  a  program  must  also  deal  with 
additional  statutory  and  other  regulatory  requirements  tied  to  acquisition  development.  Even 
if  Service-specific  requirements  (Secretary  of  the  Navy  instructions,  Army  regulations,  etc.) 
and  Defense  Acquisition  Guidebook  requirements  are  removed,  several  Title  10 
requirements  and  other  regulatory  requirements  remain  (such  as  Chairman  of  the  Joint 
Chiefs  of  Staff  Instruction  [CJCSI]  3010.02B,  3100.01  A,  3170.01  H,  3312.01  A,  6212.01  D, 
and  8501. 01  A;  DoDD  7045.20;  DoDI  4650.01, 6055.1,  and  7041.3;  and  Statement  of 
Federal  Financial  Accounting  Standards  [SFFAS]  No.  23). 

The  statutory/regulatory  documentation  breakout  resulted  in  further  decomposition  to 
identify  value-added  versus  negligible-value  or  no-value-added  documentation  (this  was  a 
qualitative  evaluation  associated  nominally  with  a  generic  Agile  software  development 
effort).  Many  documentation  requirements  have  little  or  no  value  in  supporting  a  software 
development  effort  or  the  eventual  fielding  of  software  (such  as  Programmatic 
Environmental,  Safety,  and  Occupational  Health  Evaluation,  Non-Destructive  Test  Plan,  and 
Unique  Identification  Implementation  Plan,  Failure  Modes  Effects  Criticality  Analysis, 
Performance  Based  Logistics  Business  Case  Analysis,  and  Diminishing  Manufacturing 
Sources  and  Material  Shortages);  in  these  cases,  the  PM  should  negotiate  with  the  MDA  to 
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remove  or  reduce  the  documentation  requirement,  as  appropriate.  There  are  many  cases  in 
which  the  value  of  the  document  to  the  development  effort  is  obvious,  and  program 
management  offices  should  identify  those  documents  early  in  the  program  initiation  phase  to 
ensure  proper  planning  to  accommodate  the  necessary  documentation  effort. 

A  program’s  milestone  reviews  and  documentation  streamlining  effort  can  support  a 
project’s  Agile  development;  however,  gaining  MDA  approval  for  those  efforts  can  be 
problematic  without  some  assurance  that  programs  are  still  producing  a  quality  product. 
RITE  provides  many  of  the  necessary  assurances  that  programs  need  to  gain  MDA 
approval. 


RITE  Analysis 

As  described  in  the  background  section,  the  RITE  initiative  was  created  out  of  a  need 
to  improve  the  ability  of  programs  to  meet  cost,  schedule,  and  performance  targets  of  their 
sponsors.  In  adapting  to  the  needs  of  Sec.  804,  2010  NDAA,  RITE  answers  many  of  the 
concerns  of  PMs  and  MDAs  regarding  the  rigor  necessary  to  successfully  implement  an 
Agile  development  methodology. 


In  following  the  RITE  process,  programs  use  the  RITE  Pillars  (see  Figure  7)  to  guide 
their  efforts  in  supporting  an  Agile  development  effort.  RITE  focuses  a  program’s  efforts  on 
critical  areas  proven  to  be  essential  in  successfully  developing  and  fielding  software 
products  within  cost,  schedule,  and  performance  constraints. 
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Figure  7.  RITE  Pillars 


The  RITE  process  is  not,  nor  is  it  intended  to  be,  a  panacea  for  a  program  struggling 
with  Agile  development.  It  is  intended  to  support  Agile  development  and  other  simplified, 
rapid  development  techniques  that  focus  on  product  quality  and  efficient  development. 
Combining  Agile  development  with  RITE  provides  a  program  with  the  structured  engineering 
practices  necessary  for  defense  acquisitions.  The  RITE  focus  on  contracts  is  supported  by 
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Lapham  et  al.  (2011)  in  analyzing  contracting  issues  associated  with  Agile  development: 
“Due  to  the  iterative  nature  of  Agile  and  its  propensity  to  accept  (even  welcome)  change, 
many  contracting  vehicles  present  unique  challenges  for  employing  Agile  methods.  A 
particular  issue  is  the  reporting  and  milestone  requirements  often  levied  against  DoD 
contracts”  (p.  33). 

RITE  also  includes  focus  areas  for  processes,  infrastructure,  and  organization,  which 
provide  necessary  supporting  elements  that  give  Agile  development  structure  without 
becoming  cumbersome  to  the  development  effort.  The  Process  component  of  RITE  puts  a 
greater  level  of  rigor  in  the  development  effort  and  provides  the  structure  necessary  to  keep 
Agile  development  methods  on  track.  The  Infrastructure  component  of  RITE  provides  the 
tools  necessary  to  support  Agile  development  without  hindering  flexibility;  automating  as 
much  of  the  mundane  record-keeping,  configuration  management,  and  test  tools  and  data 
ensures  that  the  development  team  stays  focused  on  development  and  not  on  writing 
reports  and  tracking  software  baselines.  The  Organization  component  of  RITE  focuses  on 
the  teaming  nature  required  in  an  Agile  development  environment.  While  it  is  common  to 
have  a  software  effort  completely  developed  by  a  contractor,  the  RITE  process  has 
identified  key  areas  in  which  government  personnel  support  development  by  integrating 
users,  developers,  and  the  integration/test  team  throughout  the  development  cycle. 

Recommendations 

Although  the  DoD  response  to  the  congressional  requirement  to  reform  the  IT 
acquisition  system  referenced  all  the  key  components  necessary  to  compel  program 
management  offices  to  consider  Agile  development  methods,  little  is  actionable  from  the 
response.  The  DoD  must  focus  efforts  on  adapting  the  DoD  5000  series  to  address 
streamlined  development  methods  and  provide  the  regulatory  authority  to  reduce 
documentation  complexity  while  maintaining  appropriate  oversight.  Pending  a  significant 
change  to  the  DoD  5000  series,  PMOs  can  still  execute  Agile  development — but  not  without 
addressing  milestone  reviews,  contracting,  and  documentation. 

The  milestone  review  process  must  transition  from  monolithic,  all-encompassing 
reviews  to  smaller,  frequent  decision  reviews  focused  on  meeting  development  targets. 
Ensuring  flexibility  in  the  process,  the  reviews  must  accommodate  changing  requirements 
and  quality  development.  The  Office  of  the  Deputy  Secretary  of  Defense  (2010)  report  to 
Congress  provides  the  basic  authority  to  execute  IT  programs  based  on  this  approach  (pp. 
9-14).  The  transition  to  frequent  decision  reviews  must  also  be  accompanied  by  a 
streamlined  documentation  effort. 

Maintaining  the  comprehensive  documentation  requirements  of  a  standard 
acquisition  program  would  severely  reduce  the  value  of  an  Agile  development. 
Documentation  should  be  focused  primarily  on  meeting  the  requirements  of  the 
development  and  sustainment  effort.  Secondary  requirements  should  include  statutory 
documentation  and  regulatory  documentation  that  cannot  be  negotiated  away.  This 
negotiation  with  the  MDA  must  be  executed  as  early  as  possible  in  the  program  initiation 
phase  as  soon  as  documentation  requirements  are  locked  down. 

Where  statutory  and  regulatory  compliance  drives  requirements  outside  the  Agile 
development  structure,  PMOs  should  ensure  that  contracts  address  those  elements  while 
maximizing  the  flexibility  necessary  to  keep  Agile  development  as  the  primary  criteria  upon 
which  the  contract  is  evaluated.  As  Lapham  et  al.  (201 1 )  noted  in  their  assessment  of  the 
value  of  implementing  an  Agile  development  methodology  to  a  PMO,  engagement  above 
the  PMO  level  is  necessary  (including  the  need  for  waivers,  mainly  from  the  MDA)  to 
address  the  departure  from  DoDI  5000.02  requirements: 
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For  example,  a  PMO  that  embraces  the  Agile  principle  that  values  operating 
code  over  extensive  documentation  may  require  a  different  set  of  CDRLs 
when  formulating  a  contract.  This  not  only  requires  a  change  in  perspective, 
but  also  the  creation  of  appropriate  governance  models,  via  tailoring  DoD 
5000.02  and  CDRLs  from  such  events  as  SRR,  PDR,  CDR,  etc.  The  PMO 
involved  may  have  to  seek  waivers  from  higher  up  the  acquisition  chain,  and 
these  higher-ups  must  also  understand  Agile  methods  if  they  are  to 
understand  what  they  are  waiving.  One  of  our  reviewers  cited  a  recent 
contract  using  Agile  methods,  in  which  they  were  bounded  by  an  SDR 
milestone,  but  obtained  approval  to  have  IDRs  (Incremental  Design  Reviews) 
beyond  that  time  instead  of  the  traditional  PDR  and  CDR  cycle,  (p.  24) 

PMOs  supporting  an  Agile  development  effort  must  work  closely  with  their  respective 
TA  to  identify  and  plan  a  successful  acquisition  strategy  that  leverages  the  best  of  Agile 
methods  while  maintaining  the  oversight  necessary  to  ensure  that  a  quality  product  is 
delivered  within  cost,  schedule,  and  performance  parameters.  The  PM  and  TA  must  present 
a  unified  front  in  gaining  approval  from  the  MDA.  The  TA,  providing  the  institutional  backing 
for  Agile  development,  should  champion  the  effort,  while  the  PM  provides  program  specific 
details  that  support  the  program’s  streamlining  requests. 

This  interaction  between  the  PM  and  MDA  is  essential  to  the  success  of  any  Agile 
development  effort  absent  significant  changes  to  current  acquisition  regulations  to  address 
the  Sec.  804,  2010  NDAA  requirements.  Implementation  of  RITE,  within  the  context  of  an 
acquisition  program’s  Agile  development  effort,  will  assist  PMOs  in  validating  and  ensuring 
compliance  with  critical  acquisition  elements,  which  is  essential  to  garner  the  support  of  the 
MDA.  RITE  is  an  Agile  enabler  for  the  government. 

Conclusion 

The  analysis  regarding  the  effort  necessary  to  streamline  a  program’s  milestone 
reviews  and  documentation  requirements  confirm  previous  research  regarding  the 
applicability  of  Agile  development  within  a  DoD  acquisition  environment.  These  results 
require  an  up-front  investment  in  time  and  effort  to  produce  a  meaningful  reduction  in  the 
milestone  review  and  documentation  effort.  PM  engagement  with  the  MDA,  in  concert  with 
the  TA,  is  essential  in  gaining  the  approvals  necessary  to  support  Agile  development.  The 
use  of  the  RITE  process  supports  the  PM’s  objective  of  creating  a  structured  environment 
that  remains  conducive  to  Agile  development  and  provides  the  MDA  with  the  comfort  level 
needed  for  approval  of  a  streamlined  milestone  review  and  documentation  effort. 
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